Method and system for processing image content for enabling high dynamic range (uhd) output thereof and computer-readable program product comprising uhd content created using same

ABSTRACT

Implementations disclosed herein (e.g., systems, methods, and computer-readable program products) provide a high definition range “UHD” compatible version of classic image content (e.g., as-released motion pictures) that was created in an era of limited dynamic range and that maintains aesthetic characterization defined by “Director&#39;s Intent” of the classic image content. Such implementations advantageously use clues to the Director&#39;s Intent found in the classic image content to make intelligent estimations as to what a Director (or other image content editing professional) was attempting to achieve in the classic image content relative to a corresponding original image content (e.g., as-shot image content). The original image content holds original imagery details that have been altered or omitted during creation of corresponding classic image content. The classic image content exhibits attributes that reflect the Director&#39;s Intent such as, for example color, contrast, vignetting, saturation, and the like.

FIELD OF THE DISCLOSURE

The disclosures made herein relate generally to processing of imagecontent such as in the form of motion pictures and still imagephotographs and, more particularly, to processing of motion pictureimage content for enabling high dynamic range (UHD) output thereof.

BACKGROUND

It is known that a considerable amount of motion picture content wascreated in a format suitable for television screens or theatricalprojection with limited brightness. Such motion picture content (andother similar forms of image content) is referred to herein as legacyimage content. Full light was typically used in creation of legacy imagecontent whereby brilliant areas of light needed to be clipped, in ordernot to darken the entire scene. Deeper in the creative process, studiolighting, as well as professional still image studio lighting, needed tofill shadows to limit the range of scenes to typically a ratio of 1:3.However, as the brightness of digital displays has increased, artistshave had wider discretion to use this wider range. This can be seen itdifferent artistic lighting of modern movies, and a slow change in theaesthetic of motion picture art.

It is also known that, in recent years, there has been a migration wayfrom fill lighting. As a result, motion picture content thatintentionally have brilliant details that require the display to hitvery high brightness levels, and intentional deep shadow detail thatrequire the display to reproduce full blacks. This type of motionpicture content (i.e., image content) is said to have Ultra HighDefinition (UHD) or High Dynamic Range (UHD).

Image content that reflects the imagery intentions of the director orother professional responsible for creating the image content (i.e.,“the Director's Intent”) is referred to herein as “classic imagecontent”. An example of classic image content is a movie (e.g., i.e.,motion picture) that has been available in the world for viewing by thegeneral public (e.g., in the form print film, DVD, etc) and that wascreated prior to the UHD capability in televisions. A common type ofclassic image content is legacy image content (i.e., legacy-type classicimage content), which was created prior to the advent of playbackequipment having UHD capability and, thus, often has low resolution andlimited detail in the highlights and shadows.

Legacy-type classic image content is created by an image masteringprofessional (e.g., a “colorist”) through manipulation of image contentthat has not yet been subjected to the Director's Intent (i.e., originalimage content). For example, the original image content may bemanipulated so that the color and illumination aesthetics are assumed tobe just right and may include other artistic manipulations such asspatial effects (e.g., vignetting) and global effects (e.g., crossdevelopment). In this respect, it is common for the legacy-type classicimage content to have been approved by the creator or director of theoriginal image content and represents the content that is intended andpreferred for public viewing.

The original image content is preferably as close to the as-capturedoriginal scene as currently exists or available. For example, theoriginal image content can be in the form of an original camera filmnegative, raw digital data, or whatever is available closest to theoriginal scene. Notably, the original image content does not bear theDirector's Intent. However, relative to corresponding legacy-typeclassic image content, the original image content does bears moreinformation in the form of resolution, detail in the highlight andshadows, and ability to recover and image with less damage. It isdisclosed herein that the original image content can be from more thanone or more sources.

Classic image content (legacy-type or otherwise) reflects the aestheticsthat has been approved by the director (i.e., represents the Director'sIntent) and that bears the color and stylistic look that would make fora preferred form of image content that is UHD compliant. This form ofimage content is referred to herein as UHD-compliant classic imagecontent. A problem arising from prior art approaches for generatingUHD-compliant classic image content is that the Director's Intent inclassic image content does not anticipate UHD imagery. Accordingly, aperfect technical UHD processing of the original image content may becompletely counter to the Director's Intent. As an example, in theclassic image content an open window may have been intended to simply bewhite and ignored, but in UHD-compliant classic image content createdusing a prior art approach, the window may become so bright with so muchdistracting detail that other action in the room is overpowered. Asanother example, in classic image content, an up-view of a man lookingdown may have intended the sky in back of the man to be soft, but in theUHD-compliant classic image created using a prior art approach, the skymay be blindingly bright such that it overpowers facial expressions andcreates a focus on the details in the sky (e.g., birds and clouds)rather than the facial expressions of the man. Thus, as shown in theseexamples, countermanding the Director's Intent, no matter how “accurate”technically, the artistic intent, and probably the storyline, is put injeopardy.

Creating UHD-compliant classic image content via known approachesinvolves manipulating corresponding original image content to createcorresponding UHD-compatible image content. More specifically, thevisual characteristic of a frame or scene being created as part of theUHD-compliant classic image was manually matched to that of acorresponding frame or scene of corresponding classic image content byadjusting a series of parameters. In this “parametric” approach, thetools available to the professional conducting the UHD processing (e.g.,a colorist) typically consist of a series of knobs (i.e., parametricknobs). These parametric knobs (e.g., each controlling on of red, green,blue, or grey brightness) are applied to the original image content inan attempt to match the color aesthetics in the classic image content.Although the professional conducting the UHD processing is assumed tohave a basic and accurate translation from a film negative to positivein place as a simulation of print film, such professional would havealso had a choice of types of film and modifiers. Because of the archaicchemical and electronic intermediates introduce many anomalies andvariables into the UHD processing, compensating for anomalies andvariables would require additional parametric knobs such as foradjusting contrast, brightness, toe and shoulder roll off, and the like.In the parametric approach, a human or a computer would adjust the knobsto get a reasonable match of the manipulated version of the originalimage content and the classic image content.

As understood by a person of ordinary skill in the art, one obviouslimitation of this parametric approach to creating UHD-compliant classicimage content is either having too few knobs or too many knobs, suchthat they become over-specified which can lead to contouring. Forexample, if an image has a continuous scale, the image shows distinctsteps when such contouring occurs (e.g., using too few bits to representan image is an easy way to introduce contouring). However, even withsufficient bits, if the representation of color intensity is notcontinuous or not continuously increasing with increasing intensity,contouring and other artifacts can occur an effect similar to banding.Another obvious limitation of this parametric approach to creatingUHD-compliant classic image content is that any spatial effects (i.e.,varying with space) such as, for example, intentional or accidentalvignetting anywhere in the original process can lead to very badresults. For example, if the center of an image was initially darkerthan the edges and the printing process darkened the edges throughvignetting, the parametric approach would choose a low contrast tobalance brightness globally, although regionally the image aestheticwould be softened by lowered contrast and would produce a very differentaesthetic than that of the Director's Intent.

Therefore, creating UHD-compliant classic image content in a manner thatcombines desirable imagery features of classic image content withdesirable imagery features of corresponding original image content andthat overcomes imagery limitations associated with known approaches forprocessing such original image content would be advantageous, desirableand useful.

SUMMARY OF THE DISCLOSURE

Embodiments of the present invention are directed to creating imagecontent that is configured for UHD playback. More specifically,embodiments of the present invention are directed to generatingUHD-compliant classic image content from corresponding original imagecontent and legacy-type classic image content. Such UHD-compliantclassic image content combines desirable imagery features of legacy-typeclassic image content with desirable imagery features of correspondingoriginal image content. Such combination is implemented in a manner thatmaintains imagery intentions of the director (or other professional)responsible for creating the image content (i.e., Director's Intent)exhibited in legacy-type classic image content with imagery obtainedfrom within the corresponding original image content from which thelegacy-type classic image content was created. In this respect,embodiments of the present invention are beneficial in recreating an UHDcompatible version of legacy image content (e.g., motion pictures) thatwas created in an era of limited dynamic range and that maintains theDirector's Intent when played back using an UHD display standard.

An object of embodiments of the present invention is to createUHD-compliant classic image content characterized by such image contentmaintaining the Director's intent of corresponding classic image contentand including aesthetic characteristics present in original imagecontent from which the classic image content was created.

Another object of embodiments of the present invention is forUHD-compliant classic image content to include feature details andaccent details from the original imagine content that are not in thecorresponding classic image content.

Another object of embodiments of the present invention is forUHD-compliant classic image content to include image content in a frameor scene in the original image content but outside of an overlappingarea of the classic image content and the original image content forsuch frame or scene.

Another object of embodiments of the present invention is for creationof the UHD-compliant classic image content to include creating a maskdefining an overlapping area of the classic image content and theoriginal image content on a per-frame and/or per-scene basis.

Another object of embodiments of the present invention is forUHD-compliant classic image content to for original image contentoutside an area of the mask to include aesthetic characterizationconsistent with the Director's Intent for image content of the classicimage content within the area of the mask.

Another object of embodiments of the present invention is for creationof the UHD-compliant classic image content to include resizing andaligning the classic image content to the original image content on aper-frame and/or per-scene basis.

Another object of embodiments of the present invention is for creationof the UHD-compliant classic image content to include subjecting classicand corresponding original image content to color match processing.

Another object of embodiments of the present invention is for colormatch processing to include parametric color match processing and pyruscolor match processing.

Another object of embodiments of the present invention is for creationof the UHD-compliant classic image content to include subjecting colormatched original image content to extended range processing.

Another object of embodiments of the present invention is for theextended range processing to provide feature details and/or accentdetails to the color matched original image content.

Another object of embodiments of the present invention is for theextended range processing to apply high frequency image content of theoriginal image content to the color matched original image content.

It is disclosed herein that the present invention can be embodied invarious forms. One such form is a method for creating UHD-compliantclassic image content. Another such form is a computer-readable programproduct having UHD-compliant classic image content created in accordancewith a method configured in accordance with the present inventionaccessible therefrom (e.g., from within memory thereof). Still anothersuch form is a system displaying UHD-compliant classic image contentcreated in accordance with a method configured in accordance with thepresent invention.

In one embodiment of the present invention, a computer-implementedmethod comprises a plurality of operations. An operation is performedfor determining an overlapping area of image content of at least oneoriginal image content source and image content of a classic imagecontent source in which visual imagery of the classic image contentaligns with visual imagery of a corresponding portion of the originalimage content. An operation is performed for performing a parametriccolor match of the original image content as a function of the classicimage content to create linear matched original image content andparametrically color matched original image content. Performing theparametric color match includes matching image content color of theoriginal image content to the classic image content as a function of acolor comparison information derived from the classic image content andthe original image content within the overlapping area. An operation isperformed for performing a pyrus color match using the parametricallycolor matched original image content and the classic image content toproduce pyrus color matched original image content. The pyrus colormatched original image content is characterized by color intensityaltered in accordance with Director's Intent of the classic imagecontent.

In another embodiment of the present invention, a method is performedfor processing image content. The method comprises a plurality ofoperations. An operation is performed for providing first image contentand second image content. The second image content is characterized by aDirector's Intent applied thereto and the second image content ischaracterized by image content thereof having color attributes alteredby the Director's Intent with respect to the first image content. Thesecond image content includes at least a portion of visual imagery ofthe first image content. An operation is performed for registering thefirst image content and the second image content to defining anoverlapping area of the first image content and the second image contentin which visual imagery of the second image content aligns with visualimagery of a corresponding portion of the first image content. Anoperation is performed for determining aesthetic characterizationdefined by the Director's Intent as a function of the second imagecontent and the portion of the first image content within theoverlapping area. An operation is performed for applying the aestheticcharacterization to an entire area of the first image content to createcolor matched original image content.

In another embodiment of the present invention, a non-transitorycomputer-readable storage medium has tangibly embodied thereon andaccessible therefrom instructions interpretable by at least one dataprocessing device. The instructions are configured for causing the atleast one data processing device to perform a method comprising aplurality of operations. An operation is performed for providingoriginal image content and classic image content. The classic imagecontent includes at least a portion of visual imagery of the originalimage content. An operation is performed for determining an overlappingarea of the original image content and the classic image content inwhich the visual imagery of the classic image content aligns with thevisual imagery of a corresponding portion of the original image content.An operation is performed for determining color attributes of theclassic image content that characterize post-processing actionsperformed with respect to the first image content. The color attributedetermining includes comparing image color information from the classicand original image contents within the overlapping area. An operation isperformed for creating color matched original image contentcharacterized by having the color attributes applied thereto. Anoperation is performed for applying extended range image content to thecolor matched original image content. Applying the extended range imagecontent includes determining color in areas of the color matchedoriginal image content that are in the saturated color condition withrespect to a color saturation range of the classic image content andadjusting color in at least a portion of the areas of the color matchedoriginal image content that are in the saturated color condition to acolor unsaturated condition.

In another embodiment of the present invention, a non-transitorycomputer-readable medium has tangibly embodied thereon and accessibletherefrom processor-interpretable information defining displayable imagecontent, the processor-interpretable information. Theprocessor-interpretable information comprises image content in a digitalformat generated using a method comprising a plurality of operations. Anoperation is performed for providing first image content and secondimage content. The second image content is characterized by a Director'sIntent applied thereto and the second image content is characterized byimage content thereof having color attributes altered by the Director'sIntent with respect to the first image content. The second image contentincludes at least a portion of visual imagery of the first imagecontent. An operation is performed for registering the first imagecontent and the second image content to defining an overlapping area ofthe first image content and the second image content in which visualimagery of the second image content aligns with visual imagery of acorresponding portion of the first image content. An operation isperformed for determining aesthetic characterization defined by theDirector's Intent as a function of the second image content and theportion of the first image content within the overlapping area. Anoperation is performed for applying the aesthetic characterization to anentire area of the first image content to create color matched originalimage content.

In another embodiment of the present invention, a non-transitorycomputer-readable medium has tangibly embodied thereon and accessibletherefrom processor-interpretable information defining a displayablevisual experience. The processor-interpretable information comprisesextended range image content jointly derived from image content of atleast one original image content source and from image content of aclassic image content source. The classic image content is a derivativeof the original image content. The extend range image content ischaracterized by extended range image content defined by a Director'sIntent of the classic image content.

These and other objects, embodiments, advantages and/or distinctions ofthe present invention will become readily apparent upon further reviewof the following specification, associated drawings and/or appendedclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

This application contains at least one drawing executed in color. Copiesof this patent or patent application publication with color drawing(s)will be provided by the Office upon request and payment of the necessaryfee.

FIG. 1 is a flow diagram view showing a method configured in accordancewith an embodiment of the present invention for creating UHD-compliantclassic image content.

FIG. 2 is a color photograph view showing conversion of original imagecontent from a negative exposure format to a positive exposure format.

FIG. 3 is a flow diagram view showing a method configured in accordancewith an embodiment of the present invention for implementing UHDprocessing of classic and original image content.

FIG. 4 is a color photograph view showing registration (e.g., alignmentand scaling) of classic image content with respect to correspondingoriginal image content.

FIG. 5A is a flow diagram view showing a method configured in accordancewith an embodiment of the present invention for implementing parametriccolor match processing.

FIG. 5B is a diagrammatic view showing a color match correlation in theform of a regression derived classic and original image content.

FIG. 6 is a color photograph view showing parametric linear fitcorrected original image content being matched to corresponding classicimage content.

FIG. 7 is a flow diagram view showing a method configured in accordancewith an embodiment of the present invention for implementing pyrus colormatch processing.

FIG. 8 is a diagrammatic view showing a representation of regional colormatch correlation.

FIG. 9 is a color photograph view showing linear matched original imagecontent and corresponding classic image content used to create colormatched original image content.

FIG. 10 is a flow diagram view showing a method configured in accordancewith an embodiment of the present invention for determining pyrus colormatch gain.

FIGS. 11A and 11B are a flow diagram view showing a method configured inaccordance with an embodiment of the present invention for implementingextended range processing.

FIG. 12 is a color photograph view showing a comparison of the extendedrange image content and pyrus color matched original image content.

DETAILED DESCRIPTION

In accordance with embodiments of the present invention, UHD-compliantclassic image content is created using corresponding original imagecontent (e.g., original camera negative (OCN), film intermediate,digital intermediate which has an extended range or more than one sourceof original image content.) and legacy-type classic image content thatwas created from the original image content. One example of OCN is colornegative film, which generally exhibits full dynamic range, color gamut,and other “accuracy” aspects (i.e., beneficial exposure characteristics)such that a full range of an original scene(s) in the original imagecontent can be sufficiently replicated (e.g., digitally) and displayedwith technical fidelity. In any case, preferably, the original imagecontent utilized in embodiments of the present invention is as close aspossible to the originally-captured scene (i.e., as-filmed scene) ascurrently exists or available. The key attribute of the original imagecontent is that it holds imagery details of the originally-capturedscene (i.e., original imagery details) that have been altered or omittedduring creation of corresponding classic image content (e.g., creationof classic image content from original image content correspondingthereto).

In contrast to the original image content, classic image contentexhibits attributes that reflecting the Director's Intent. Legacy-typeclassic image content will generally lacks attributes such ashighlights, shadows, smooth grain, and resolution detail available froma corresponding OCN (i.e., original image content), However, legacy-typeclassic image content will preferably have color, contrast, vignetting,saturation, and other imagery details defined by the Director's Intent(director's imagery details).

High Dynamic Range Aesthetic Match

It is not possible to know exactly what a Director (i.e., a classicimage content creating professional) would have done in creatinglegacy-type classic image content from corresponding original imagecontent had that Director been privy to UHD capabilities. However,embodiments of the present invention advantageously use clues to theDirector's Intent found in the legacy-type classic image content to makeintelligent estimations as to what the Director was attempting toachieve in the classic image content relative to the correspondingoriginal image content. Such intelligent estimations are referred toherein as UHD aesthetic match processing. To that end, embodiments ofthe present invention are configured for creating a new instance ofimage content (i.e., UHD-compliant classic image content) characterizedby classic image content that is provided with UHD attributes and thatmaintains the aesthetic characterization defined by the Director'sIntent. One example of such aesthetic characterization includesretaining all basic color correction, reproducing luminance in largeareas and reproducing color intensity (highlights) in areas of limitedrelative size. For example, an open window and background sky in a scenewould have the same (or approximately the same) brightness that it wasin the classic image content, all unsaturated colors and contrasts wouldmatch the classic image, and from a distance, an audience wouldexperience the luminance of the classic image. Another example of suchaesthetic characterization includes restoring color to saturated areas.Thus a saturated sky or an open window should reveal natural blue orgolden light. Taking a clue from classic painters through the centuries,it may be preferred for color saturation to be less than 100% (e.g., hueand more than half the chroma intensity relative to the luminance).Another example of such aesthetic characterization includesre-introducing accent details (e.g., lights in eyes, shimmer fromsequins on dresses, luminary points like holiday lights, sparkles fromjewelry, flashes of light from explosions and the like) that weresaturated out during creation of the classic image content. It ispreferred to see the colors and brightness of such accent details, bitin balanced ratio and/or proportion to the newly created image content.Another example of such aesthetic characterization includesre-introducing feature details (e.g., foreground features, backgroundfeatures, skin definition, and the like) at were saturated out duringcreation of the classic image content. Preferably, these feature detailswill be re-introduced with the same or similar brightness as in theclassic image content (e.g., about half in areas of the classic imagethat have saturated completely). Thus, a generalized characterization ofthe aesthetic characterization is for the regional brightness of theclassic image to be matched, for accent details from the original imageto be reintroduced, for color of the classic image to be maintained orreintroduced to at least about half the level of the original imagecontent, and for feature detail from the classic image content to bemaintained and to be at least partially reintroduced from the originalimage content.

These and other goals of UHD aesthetic match processing can beaccomplished by embodiments of the present invention by the followingprocess operations (i.e., UHD aesthetic match process). For a givenscene or series or scenes, classic image content and original imagecontent (e.g., as defined above) are provided. The classic image contentand the original image content are processed with a “sharp-unsharp mask”to produce “classic-derived image content” and “original-derived imagecontent”, which jointly define the underlying basis (i.e., code to life)for the Director's Intent. Preferably, the sharp/unsharp mask is ablurring filter that maintains the sharpness of strong edges whileblurring weak detail. For example, the sharp/unsharp mask can be similarto a well-known “surface blur” in which integrated weighting is afunction of distance as well as pixel count level spread. It can beimplemented as a surface blur in a pyramid (e.g., an iterative series offrequency levels) such that strong impulses are also blurred if theycomprise a small area. It is disclosed herein that a sharp-unsharp maskin accordance with embodiment of the present invention is used as acontrol in many commercial implementations of shadow fill, such asDigital SHO and Adobe Photoshop (i.e., via menu selection ofImage>Adjustments>Shadows/Highlights). After the classic-derived imagecontent and the original-derived image content are generated, theclassic-derived image is subtracted (e.g., numerically) from the classicimage content and the original-derived image is subtracted (e.g.,numerically) from the original image content, thereby respectivelyproducing “classic image characterizing content” and “original imagecharacterizing content”. Next, the classic-derived image is combinedwith the original image characterizing content image to create resultingUHD-compliant classic image content, which is an image combining theartistic aesthetic of the original image content plus the aesthetic ofmodern UHD image content.

A variation of the UHD aesthetic match processing is to combine apre-selected or controlled proportion of the derived image and imagecharacterizing content versions of the classic and original imagecontent with such proportions being greater than 0% and less than 100%.For example, some of the original image characterizing content can becombined with proportionally less of the original image characterizingcontent to form the UHD-compliant classic image content. In thepreferred embodiment, this combining is nonlinear so as to be themaximum of the original image characterizing content and a selectedproportion of the original image characterizing content. Thus, theoriginal image characterizing content acts as a “safety net” for detailthat was lost to saturation.

A further variation of the UHD aesthetic match processing firstseparates the various image content instances into aluminance-chrominance space such as YUV or Lab, and selects differentproportions as described above for the luminance channels than for thechrominance channels. YUV refers to a color space in which “Y” is theluminance (brightness) channel and “UV” is the chrominance (color)channels. In particular, an increased amount of the original-derivedimage content and proportionally less of the classic-derived imagecontent is combined for the chrominance channels to form theUHD-compliant classic image content. In the preferred embodiment, thiscombining is nonlinear and is the maximum of the classic-derived colorand a selected proportion, typically more than half, of theoriginal-derived color.

In a further variation of the UHD aesthetic match processing, theoptimum ratio of mixture is established, and images calculated for, twodisplays with different levels of UHD. The difference between the two isfurther compressed and transmitted with the main image as a smallsidecar image. At reception, select proportions of the sidecar image areapplied to the main image to produce optimum viewing for a range ofdisplays under a range of ambient lighting conditions.

Color Match Using Detail Tracking

As previously discussed, classic image content is typically approved bythe creator-director of such content and has color aesthetics andartistic manipulations (e.g., spatial effects like vignetting and globaleffects like cross development) that are assumed to represent theDirector's Intent. In this respect, legacy-type classic image contentfor a given title is a common form of image content that the public hasbecome used to seeing and, for archival and curation purposes, it is theform of image content that must be matched when creating a UHD-compliantversion of such legacy-type classic image content.

It is well known that legacy-type classic image content often hasserious artifacts. This is because it is common for legacy-type classicimage content to have been taken from an aged video stream. As a result,it is not uncommon for such image content to have low resolution andeven lower chromatic resolution, to have electronic artifacts such as“ringing”, and to have digital compression artifacts that remove subtledetail. Additionally, it may be from a multi-generational print in orderto get through archaic chemical editing processes to a distributionprint, whereby it has low resolution, graininess, and embedded defects(e.g., dust and scratches). In summary, the “classic” image is a belovedmemory that is definitely not ready for modern ultra-high-resolutioncommercial distribution.

As previously discussed, original image content (e.g., original cameranegative (“OCN”)) has imagery details that corresponding legacy-typeclassic image content lacks. In this respect, as compared tocorresponding legacy-type classic image content, the original imagecontent has higher resolution, less graininess, fewer artifacts, and farmore accent and feature detail. The original image content, however, hasnone of the manipulations characterizing the Director's Intent inaddition to its brightness and color varying greatly.

Advantageously, embodiments of the present invention preferablyimplement color match processing. The objective of such color matchprocessing is to create a combined version of legacy-type classic imagecontent and corresponding original image content, which closely(preferably exactly) matches the aesthetics of the classic image contentand that has full resolution, beautiful, modern image. Ideally, colormatch processing would include extracting UHD detail from the originalimage content and merging it with the legacy-type classic image contentto form resulting reconstructed UHD image content (e.g., UHD-compliantclassic image content). Examples of such UHD detail include, but are notlimited to, adding color in saturated areas, adding accent details, andadding feature details. The objective is for the UHD-compliant classicimage content to have an as-viewed (e.g., from a distance) characterthat exactly replicates that of the legacy-type classic image content.All colors and all spatial effects, such as vignetting, transferperfectly because the low frequencies of the reconstructed image arecopied directly from the classic image.

As a skilled person would appreciate, such an ideal implementation ofcolor match processing would not work in practice for several reasons.For example, this ideal implementation of color match processing assumesthat a magnitude of high spatial frequency detail in the original imagecontent matches what is needed in the legacy-type classic image content.As skilled person would recognize that this is often not the case.

In practice, aesthetic detail in legacy-type classic image content andcorresponding original image content can be far different from eachother depending on local contrast in each. Moreover, the contrast in thelegacy-type classic image content and corresponding original imagecontent may vary a lot across the gray scale and therefore across theimage spatially, and may even approach zero in saturated areas. Stillfurther, such aesthetic detail will vary differently for differentcolors.

Advantageously, embodiments of the present invention provide forpracticable implementation of color match processing. Such embodimentscan overcome the aforementioned issues by measuring and comparing theintensity of spatially local markers that appear in both of the imagecontents (i.e., classic and original), and use the ratio of theintensity of these markers to predict a gain that should be applied tothe high frequencies of the original image content to equal the missinghigh frequencies of the classic image content. As an example, theintensity of the middle spatial frequencies, which exist in both of theimage contents, is used as a regional predictor of how the high spatialfrequencies should be modified to match those lost from the classicimage. Beneficially, contrast and cross colors affect middle spatialfrequencies in the same manner as high spatial frequencies. Thus, oncethe magnitude of the high spatial frequencies of the original imagecontent has been regionally corrected to the classic image content, thehigh frequencies are added into the classic image content to formresulting reconstructed UHD image content (e.g., UHD-compliant classicimage content).

As a refinement to color match processing, the classic image content maybe further band limited to reduce the crossover point. This can bebeneficial because, in many cases, the highest frequencies of theclassic image content may contain electronic artifacts, surface defects,and unpredictable variability. In a preferred embodiment, the absolutevalue of the horizontal first derivative of the original image contentis added to the absolute value of the vertical first derivative of theoriginal image content, thereby forming a magnitude original. The sameoperation is performed on the classic image content to form a magnitudeclassic. This method has the advantage of enabling centering magnitudeof high frequency details on actual edges. It can also be desirable toapply a wide blur and fade filter whereby regions with very littlesignal will inherit the characteristic of nearby areas of morecertainty. This can be followed by an even wider blur and an even morecomplete fade back. This can be done in other ways with the ultimatepurpose being so that an area of the image with very little detail willinherit the characteristic of surrounding areas rather than suffer froma wide uncertainty of noise.

With respect to chromatics, it is disclosed herein that processing athree-color image as three separate independent monochrome channelsworks very well. Although the cross-color gain at the high frequenciesmay be different, the human eye has low sensitivity to high spatialfrequencies of the color channel. As such, the subtle differences in thecross-color gain at the high frequencies are typically invisible to thehuman eye. To perfect high frequency crossover, the two classic andoriginal image content can be preprocessed into a luma-chroma colorspace such as YUV, thereby allowing the then the Y and UV images to beseparately processed. The chroma-channel gains may be guided in weakareas by luminance gain because the chroma-channels may be subject tohigher noise.

If the classic image is from an electronic form (e.g., DVD, tape or NTSCLaser Disc), then the chroma-channels may be much lower resolution thanthe luminance-channel. For example, with respect to NTSC Laser Disc inNTSC the Q color channel may exhibit more than 8 times lower horizontalresolution than the Y luminance channel. This can be managed in severalways. A preferred way is for the original to perform the sameasymmetrical chroma blur on the original image in the production ofblurred-original. Another option is to preprocess into a matchingluma-chroma color space as above, such as YUV or YIQ. Then the threechannels are processed as before but with lower transition frequenciesfor the chroma-channels.

Automated UHD Processing

In preferred embodiments, ultra high definition (UHD) processing ofimage content includes UHD aesthetic match processing and color matchprocessing, both of which hare discussed above. The resultant of UHDprocessing is UHD-compliant classic image content. The UHD-compliantclassic image content exhibits ultra high definition when displayed(e.g., when played back). Creation of the UHD-compliant classic imagecontent is implemented using high resolution, high information contentthat is at or near an original source of image content (i.e., originalimage content having high resolution and dense imagery information) incombination with image content representing a particular aestheticrendering of a Director or the like (i.e., classic image contentcharacterized by Director's Intent).

As shown in FIG. 1, there are several steps in the automated processingto merge the classic image content and the original image content. Theoutput of the automatic UHD processing provides image content offeringultra high definition appearance during playback (i.e., UHD-compliantclassic image content). If additional adjustments are desired, preferredembodiments of the present invention are configured for allowing suchadjustments to be selectively made by, for example, an image processingprofessional (e.g., a colorist).

Still referring to FIG. 1, a method configured in accordance with anembodiment of the present invention for creating image content offeringultra high definition appearance during playback (i.e., UHD-compliantclassic image content) is shown. Classic image content (e.g.,legacy-type) and original image content from which the classic imagecontent was derived serve as input image content (blocks 102 and 104,respectively). Preferably, the original image content utilized is asclose as possible to the originally-captured scene (i.e., as-filmedscene) as currently exists or available. The key attribute of theoriginal image content is that is holds imagery details of theoriginally-captured scene (i.e., original imagery details) that havebeen altered or omitted during creation of corresponding classic imagecontent (e.g., creation of classic image content from original imagecontent corresponding thereto). In contrast to the original imagecontent, classic image content exhibits attributes that reflecting theDirector's Intent. Legacy-type classic image content will generallylacks attributes such as highlights, shadows, smooth grain, andresolution detail available from a corresponding OCN (i.e., originalimage content), However, legacy-type classic image content willpreferably have color, contrast, vignetting, saturation, and otherimagery details defined by the Director's Intent (director's imagerydetails). Accordingly, the original image content bears more informationin the form of resolution, detail in the highlight and shadows, andability to recover and image with less damage compared to the classicimage content whereas, in contrast to the original image content, theclassic image content includes imagery details defined by the Director'sIntent.

For original image content that is a film format, a scanning operationis performed (block 106) for providing the original image content in asuitably configured digital file format (block 108). Preferably, imagecontent in a film format is scanned with a high-resolution scanner thatoutputs a digital file at or above the resolution of original contentformat. Examples of suitable commercially available scanners include,but are not limited to, those offered under the brands ARRI,Lasergraphics and Northlight by their respective company(ies).Furthermore, additional cleanup of the original image content (now indigital file format at block 108) can be performed with one or morecommercially-available tools (block 110). For example, additional defectremoval can be performed using one or more commercially available toolsoffered under the brand Color ICE and Black ICE by their respectivecompany(ies).

In some instances, as shown in FIG. 2, providing the original imagecontent in a suitably configured digital file format will requireconverting the original image content from being in a negative exposureformat (NE) to a positive exposure format (PE). The polarity(positive/negative) of the original image content and of the classicimage content, which is always positive, needs to be the same. Thus, ifthe original image content is negative, it will need to be inverted andmade into positives. For example, the negative-to-positive inversion canbe defined as one over the original pixel values raised to a power andscaled by maximum value. Optionally, the transfer curve of the originalimage content can be generally matched to the classic image content, forexample, by changing the gamma, or apply a different (e.g.,user-defined) transfer curve. This does not need to be precise, and canbe done many ways as known in the art.

Next, the original image content that has been suitably pre-processedinto a digital form (block 112) and the classic image content (block102) are subjected to UHD processing (block 114) for producing imagecontent offering ultra high definition appearance during playback (block116). Such image content offering ultra high definition appearanceduring playback is also referred to herein as an UHD-compliant classicimage content (i.e., extended range image content). The UHD-compliantclassic image content is a restored version of the original imagecontent with the color and intensity of the classic. Advantageously, asdiscussed below in greater detail, the UHD-compliant classic imagecontent also maintains all the resolution and information content of theoriginal image content and has an area greater as large as the originalimage content (i.e., area of the restored version of the original imagecontent extends beyond the area of the mask). In this respect, theUHD-compliant classic image content is a newly created instantiation ofimage content characterized by classic image content that is providedwith UHD attributes and that maintains the aesthetic characterizationdefined by the Director's Intent and, thus, combines desirable imageryfeatures of the classic image content with desirable imagery features ofthe corresponding original image content

If it is determined that the UHD-compliant classic image contentrequires additional touch-up, such additional touch-up can beimplemented accordingly (block 118) such as, for example, by a coloristto produce UHD-compliant classic image content so adjusted (block 120).After such additional touch-up or if such additional touch-up is notrequired, the UHD-compliant classic image content is subject to finalformatting (block 122) to produce a ready for distribution version ofthe UHD-compliant classic image content (block 124).

The UHD-compliant classic image content (i.e., the output from the UHDprocess) is a full resolution series of digital frames. For commercialuse, this UHD-compliant classic image content is preferably encodedusing industrial standards and formatted for a device to be used forplayback of the UHD-compliant classic image content. Examples of suchstandards include, but are not limited to, perception quantizer (PQ)curve, International Telecommunications Union's (ITU) recommendationBT.2020 (i.e., Rec. 2020) and High Efficiency Video Coding (HEVC) series265 (i.e., H.265).

Referring now to FIG. 3, in a preferred embodiment, UHD processing(block 114 in FIG. 1) begins with frames of the original image content(block 130) that match the frames of the classic image content (block132) being selected (block 134), thereby producing matching classicimage content frames (136) and original image content frames (138). Suchframe selection is required because there are often multiple takes foreach scene in the original image content. In addition, only part of theselected take may be cut and used for the classic image content. It isdisclosed herein that the original and classic image contents can bematched on a per-frame and/or per-scene basis. Image content of a frameor a scene is referred to herein as visual imagery.

It is disclosed herein that the original image content cam be from oneor more original image content sources. For example, a first originalimage content source can be an original camera negative of a backgroundand a second original image content source can be a ‘green screen’media. In this respect, a plurality of original image content sourcesjointly define the original image content.

The classic image content (e.g., frames thereof) and the original imagecontent frames (e.g., frames thereof) are subjected to one or moreoperations for registering each frame or scene in the classic imagecontent with a respective frame or scene of the original image content(block 140) to produce classic image content that is framed/aligned(block 142) with respect to the original image content and a mask (block143) defining the overlapping area of the classic and image contents(e.g., matching visual imagery of frames and/or scenes thereof). Oneoperation associated with such registration involves the classic imagecontent (e.g., frames thereof) being cropped for formatting and/or forDirector's Intent compared to the original image content frames (e.g.,frames thereof). Another operation associated with such registrationinvolves color, dynamic range and content of the classic image contentbeing modified for the Director's Intent. Still another operationassociated with such registration involves the classic image contentbeing re-formatted and compressed for the intended media. As shown inFIG. 4, yet another operation associated with such registration involvesthe classic image content and original image content being scaled thesame and aligned on a per-frame and/or per-scene basis to define arespective image content mask (M) that defines a boundary of theoverlapping area of the classic and original image contents.

The classic image content has an equal or lower resolution compared tothe original image content and, typically, has had additional processing(sharpening, compression, etc). The classic image content is often acropped from the original image content so it contains a smaller area ofthe original image content. In some cases, an area encompassing visualimagery of the original image content comprises only a portion of thevisual imagery of the classic image content (e.g., the classic imagecontent comes from a plurality of sources). The colors in the classicimage content are also different from the original image content such asdue to the Director's Intent. Accordingly, given classic image contentthat can be very different in resolution, information content, color andarea from the corresponding original image content, the classic imagecontent typically needs to be aligned and warped to match the originalimage content spatially to be useful as an input in UHD processing. Inaddition, regions that are common to both the original image content andthe aligned classic image content (i.e., image content mask) need to beknown. For example, the original image content usually contains imagecontent beyond the boundaries of the classic image content, andsometimes a “green screen” dropout has been used to substitute foreignimage into the classic image content that is not in the original imagecontent, and therefore, the areas of the green screen are not common.

Still referring to FIG. 3, UHD processing (block 114, also in FIG. 1)configured in accordance with embodiments of the present invention caninclude parametric color match processing (block 144) that determinesparameters of an equation relationship between the colors andintensities of the classic image content compared to the original imagecontent. A primary purpose of the parametric color match processing isto approximate close match of the colors (and their respectiveintensities) of the classic image content (i.e., Director's Intentcolor) in the original image content based on the area of the classicimage content and original image content that is common therebetween(i.e., in the case where the classic image content is cropped fromwithin the original image content).

Output of the parametric color match processing includes original imagecontent that has been parametrically matched to color in the classicimage content (block 146—i.e., parametrically color matched originalimage content) and original image content that has been linear matchedto color in the classic image content (block 150—i.e., linear colormatched original image content). The parametrically color matchedoriginal image content is characterized by having color thatapproximates that of the classic image content and that has accent andfeature details in areas corresponding to those of the classic imagecontent that are unsaturated. In this respect, the parametrically colormatched original image content is a version of the original imagecontent that has color that closely approximates that of the classicimage content and that has accent and feature details corresponding tothat in areas that are in the saturated range of the classic imagecontent (i.e., saturated areas in the classic image content are alsosaturated in the parametrically color matched original image content).In contrast, with respect to the linear color matched original imagecontent, areas of the original image content that correspond tosaturated areas of the classic image content are unsaturated accordingto a linear regression (i.e., a regression) between the original andoriginal image contents. For example, as discussed below in greaterdetail with respect to FIG. 5A, a linear regression (i.e., an example ofcolor comparison information) is developed from a scatter plot of colorintensities between the classic and original image contents. This linearregression is used to adjust color intensity of unsaturated areas of theoriginal image content to color intensities dictated by the linearregression of the classic and original image contents. In this respect,the linear color matched original image content carries the aestheticcharacterization of the classic image content into areas of the originalimage content that are saturated in the classic image content.

It is disclosed herein that parametric color match processing can beimplemented frame-by-frame, or scene-by-scene, or an obvious combinationof frame-by-frame and scene-by-scene. For example, in a panned scene,when the camera passes through a low information segment, it can bepreferable to base parametric color match on frames before and after thelow information segment.

Still referring to FIG. 3, UHD processing (block 114, also in FIG. 1)configured in accordance with embodiments of the present invention caninclude pyrus color match processing (block 152) and extended rangeprocessing (block 154). The pyrus color match uses a multi-frequencylevel blurring and the mask (see block 143 in FIG. 3) to determine thebest information to use for the particular pixel in determining thecolor match.). In this respect, the ratio of the colors of the classicimage content in comparison to the original image content outside theoverlapped area are estimated based on the proximity of the pixel to theoverlapped area. The pyrus color match processing, which has theparametrically matched original image content (block 146) and classicimage content (e.g., block 142) as its input, allows for colorvariations across a scene (or frame) and beneficially accommodatesvisual color fit between the two image contents than the parametriccolor match processing is capable of creating. The classic image contentis used as a source for highlight details within unsaturated areasthereof. These highlight details (e.g., feature and accent details) arealso present in the original image content. Original image content thathas been parametrically and pyrus color matched to the classic imagecontent is the output of the pyrus color match processing (block156—i.e., pyrus color matched original image content). Such output is aversion of the original image content that matches the color attributes(e.g., Director's Intent) of the classic image content with a highdegree of accuracy (i.e., markedly higher than the output of theparametric color match process) but without feature and accent detail inareas that correspond to areas of the classic image content withsaturated colors. Such feature and accent detail is otherwise present inthe unprocessed original image content (e.g., block 104 in FIG. 1), buthas been lost during the parametric color match process as a result ofapplying the Director's Intent to the original image content.

The extended range processing, which has the pyrus color matchedoriginal image content (block 156) and the linear color matched originalimage content (block 150) as its input, makes use of information withinthe linear color matched original image content to create higher dynamicrange than is available in the classic image content without losingimagery details defined by the Director's Intent. Output of the extendedrange processing is extended range image content (block 116; see also inFIG. 1) that combines desirable imagery characteristics of the classicimage content with desirable imagery characteristics of correspondingoriginal image content processing is the output of the extended range(i.e., UHD-compliant classic image content also referred to herein asthe extended range image content). More specifically, the output of theextended range processing is a version of the original image contentthat identically or nearly identically matches the color attributes(e.g., Director's Intent) of the classic image content and that hasfeature and accent detail in areas that correspond to areas of theclassic image content with saturated color.

As discussed above, the pyrus color matched original image contentmatches the color attributes (e.g., Director's Intent) of the classicimage content with a high degree of accuracy (i.e., markedly higher thanthe output of the parametric color match process) but without featureand accent detail in areas that correspond to areas of the classic imagecontent with saturated colors (i.e., this feature and accent detail isotherwise present in the unprocessed original image content—e.g., block104 in FIG. 1). Advantageously, extended range processing in accordancewith embodiments of the present invention serve to recapture color andhighlights in areas of the original image content that have becomesaturated during parametric and/or pyrus color match processing. Inpreferred embodiments, extended range processing is implemented in amanner that recaptures color and highlights while preserving theoverarching aesthetic character of the Director's Intent. To this end,recapture of such color and highlights can be implemented in a mannerwhereby the degree of recapture is inversely related to a unit area of aparticular portion of the image content. For example, deriving extendedrange image content can include determining a proportionality forunsaturating color in portions of the pyrus color matched original imagecontent that are within the color saturation range of the classic imagecontent, wherein the proportionality for unsaturating color for aparticular portion of the pyrus color matched original image content isperformed as a function of degree of saturation of the particularportion and a relative size of the particular portion with respect toadjacent portions thereof. In one example of such extended rangeprocessing, a window and light shining through it has been intentionallysaturated so as to mute the presence of the window and light relative toa person standing in front of the window (i.e., make the person thefocal point of the scene). In such case, color and highlights for suchparticular portion of the image content is recaptured but to arelatively low degree because the window and light shining through itmake up a substantial portion of an encompassing area. However, inanother example of such extended range processing, edges of sequins on adress and other nearby shiny objects make up a relatively small portionof an encompassing area. As such, color and highlights for suchparticular portion of the image content is recaptured to a relativelyhigh degree because the sequins and other nearby shiny objects make up arelatively small substantial portion of an encompassing area.

Turning now to a discussion of the parametric color match processing(block 144 in FIG. 3), a detailed process for implementing suchparametric color match processing is discussed in reference to FIG. 5A.The parametric color match processing assumes an equal relationshipacross each of the image contents (i.e., classic and original) for eachcolor and determines an average relationship. If a region of one of theimage contents is different from the other, the average relationshipwill not capture the difference. The quality of the fit depends on thenumber of parameters, for example lightness, color, color intensity inall 3 dimensions, shadow and highlight saturation, and so forth. Not allof these parameters can be fully solved in all frames that lack parts ofthe gray scale. Accordingly, parametric color match processing is bestdone over a larger number of frames and fewer parameters than can besolved unambiguously. In preferred embodiments, parametric color matchprocessing is used to get close to a suitable set of color matchparameters and then the pyrus color match processing (block 152 in FIG.3) is implemented for matching color between the image content to a finelevel of resolution in a manner that overcomes the limits of parametricmatching and that provide visually perfect (or near perfect) matching.The pyrus color match processing is discussed below in greater detail inreference to FIG. 7.

Still referring to FIG. 5A, a specific implementation of the parametriccolor match processing (block 144) is disclosed. Such specificimplementation begins the original image content (block 132; see also inFIG. 3), which is in or is processed (block 202) to be within a positiveexposure format (as discussed in greater detail above in reference toFIG. 2), being blurred (block 204) to match a corresponding portion ofthe classic image content (block 134, see also in FIG. 3). This blurredoriginal image content (block 208) is then compared in the overlappingarea against the corresponding portion of the resized and registeredclassic image content (block 140; see also FIG. 3) to create a virtualscatter plot (block 212). A weighted linear regression (block 214) usesdata from the scatter plot and from weighting indicating colorintensities in the classic image content (block 216), followed byadjusting (block 218) in accordance with the linear regression to createlinear matched original image content (block 150, see also in FIG. 3).The linear regression can be used to define a linear relationship ofcolor of the original image content to the classic image content (i.e.,color comparison information) such as, for example, via creation of ascatter plot (blocks 222, 224). In preferred embodiments, the linearregression specifies 6 color match parameters, which are level and slopeof red, green, and blue. Embodiments of the present invention canfurther identify highlight and shadow regions in each of the colors toproduce 6 or more additional parameters (e.g., 12 total color matchparameters). It is disclosed herein that higher order and cross terms inthe regression can create a better fit, but at the expense of potentialerrors in image content with limited greyscales.

Saturation matching (block 226) is performed using data from the scatterplot (block 224) of the linear matched original image data (block 150)followed by saturation adjustment (block 228) being performed on thelinear matched original image data (block 150). In this respect, thelinear terms are adjusted first and the highlight and saturation termsare found and applied to the adjusted original image content.Accordingly, dependent upon the aforementioned twelve (12) color matchparameters, the original image content is used to create theparametrically color matched original image content (block 146, see alsoin FIG. 3). The underlying objective of saturation matching andsaturation adjustment is applying Director's Intent with respect tocolor saturation. To this end, portions of the linear matched originalimage content having color that is in a color saturation range of theclassic image content are determined and adjusting the original imagecontent portions to have a degree of saturation as a function of thecolor saturation range of the classic image content (e.g., a linearregression derived as a function of the linear matched original imagecontent and the classic image content).

Referring to FIG. 5B, a color match correlation 250 is shown. The colormatch correlation 250 is in the form of a regression derived fromclassic and original image content. For example, the regression can bederived from a scatter diagram of pixel-by-pixel of color intensitycomparison. The color match correlation 250 includes a parametric colormatch curve 252 and a linear color match curve 254. The linear colormatch curve 254 includes a linear portion 256 of the parametric colormatch curve 252 with linear extension above and below upper and lowerlimits of such linear portion of the parametric color match curve 252.As disclosed herein, the parametric color match curve 252 and the linearcolor match curve 254 are utilized in adjusting color intensity of theoriginal image content as a function of the classic image content tocreate respective color matched versions of the original image content.

FIG. 6 shows an example of parametric linear fit corrected originalimage content being matched to corresponding classic image content, suchas provided for by parametric color match processing discussed above inreference to FIG. 5A. A non-saturating parametric relationship of thecolor space of the original image content, compared to the classic imagecontent, is determined for the overlapping areas, and this equation isused to adjust the colors of the original image content to approximatelymatch the color of the classic image content in a global sense. This istypically matched across the middle parts of the gray scale to avoidanomalies caused by saturation at the ends of the gray scale. Theimplementation as disclosed is purely linear, i.e. brightness and slope,however other higher order terms can be added, such as gamma.

Turning now to a discussion of the pyrus color match processing (block152 in FIG. 3), a detailed process for implementing such pyrus colormatch processing is discussed in reference to FIG. 7. Pyrus color matchprocessing allows for color variations across image content (e.g., ascene thereof). Pyrus color match processing also accommodates a bettervisual color fit between original and classic image content thanparametric color match processing is capable of creating. Regional colormatching works best when the differences between the classic andoriginal image contents are first minimized (preferably globallyminimized). The linear color match is preferably also used with theregional color match for creating the extended range image content.

Referring to FIG. 7, the parametrically matched original (block 146: seealso in FIGS. 3 and 5) is blurred (block 302) to match the classic imagecontent (e.g., a frame or scene thereof) and is blurred (block 304) toremove noise, resulting in a low-pass blurred version (e.g., via a lowfrequency filter) of the original image content (block 306), whichcreates lower frequencies or low pass of the parametrical match in colorbetween the classic and original image contents. The lower frequenciesproduce an image that is a better match to the low pass classic,eliminate noise, and avoid high frequency information. This low passimage is passed into the pyrus blur operation (block 312) for creating acorresponding bandpass image (i.e., mid frequencies).

The resized and registered classic image content (e.g., a frame or scenethereof) with defined edges (block 148; see also in FIG. 3) is alsoblurred (block 308) to remove noise, resulting in a low-pass blurredversion (e.g., via a low frequency filter) of the classic image content(block 310). The low-pass blurred version of the classic image content(block 310) exhibits lower frequencies or low pass of the classic imagecontent. The lower frequencies produce an image that eliminates noise,and avoids artifacts. This low pass image is passed into the pyrus bluroperation (block 314), which creates a bandpass image (i.e., midfrequencies).

The low pass original image content and the classic image content areboth subjected to pyrus blur operations (blocks 312, 314, respectively)using a mask (block 143; see also in FIG. 3) that defines theoverlapping area of the original and classic image contents, therebyproducing a pyrus blurred version of the classic image content (block316) and a pyrus blurred version of the classic image content (block318).

Each one of the pyrus blur operations includes a series of blur actionsapplied to the respective image content and the mask at successivelylower frequencies. The resulting blurred image content is divided by theresulting blurred mask, thereby allowing the image content beingextended beyond the mask boundaries (i.e., classic image content beingextrapolated to the boundaries of the original image content). Theresolution or frequency can decrease gradually beyond the maskboundaries. It is disclosed herein that the lower frequency componentcan carry the information farther the from the mask boundaries.

Parameters defining the pyrus blur operations (blocks 312, 314) for theoriginal image content and the classic image content are used in anpyrus color match gain process (block 320) for determining an pyruscolor match gain (block 322). This pyrus color match gain is applied tothe difference (block 324) of the parametrically color matched originalimage content (block 146) and the pyrus blurred original image content(block 316), with a resultant thereof being added (block 325) to thepyrus blur of the classic image content to create a version of theoriginal image content with the color and intensity of the classic(block 326). In this respect, the low pass parametrically matched image(block 146), the low pass classic image (block 148), and the mask (block143) are used to determine the conversion or gain parameters to get thecolors of the parametrically matched original image content toregionally match the classic image content whereas the parametric colormatch was implemented for globally match the classic image content. Thelow pass images from the blur operations are used to create band passimages to determine the conversion parameters regionally. The regionallycolor matched image is used in the linear region of the parametric matchand, in the saturation regions, the pixel intensities are determined byextending the parametric match beyond the saturation intensities in eachchannel.

Preferably, all computations are performed in floats (or doubles) sothat no data is saturated and so that minimal quantization occurs.Advantageously, the restored version of the original image content alsomaintains all the resolution and information content of the originalimage content and has an area greater than the classic image content andas large as the original image content (i.e., area of the restoredversion of the original image content extends beyond the area of themask).

FIG. 8 shows a representation of regional color match correlation 360.The regional color match correlation 360 allows for a higher level ofprecision in color match of the original image content 362 to theclassic image content 364 via the pyrus color match. Specifically, theparametric color match adjusts color characteristics of the originalimage content 362 as a function of a global color match correlation(e.g., the parametric color match curve 252 in FIG. 5B) therebyproducing a globally color matched version of the original imagecontent. The global color match correlation is derived from anoverlapping area 366 of the classic image content 364 and original imagecontent 362. In contrast, the pyrus color match is performed foradjusting color characteristics of each one of a plurality of differentregions of the globally color matched original image content as afunction of a respective one of a plurality of regional color matchcorrelations. Each of the regional color match correlations is derivedfrom a corresponding region of the overlapping area 366 of the classicimage content 364 and original image content 362. For example, as shownin FIG. 8, the closer a pixel is outside the overlap boundary 364, themore high frequency information it takes from the overlap area 366. Inthis respect, if the pixel is just outside a boundary 364 of theoverlapping area 366 (e.g., pixel 370), the estimate for that pixel isderived from the pixels just inside the overlap area 366. If the pixeloutside the overlap area 36 is far from the boundary (e.g., pixel 368),the estimate is mostly derived from the lower frequency blur componentsof the overlap area 366. In a preferred embodiment, being defined by therespective position of the particular region refers to being defined bya distance from the particular region outside of the overlapping area363 to the boundary of the overlapping area.

FIG. 9 shows linear matched original image content and correspondingclassic image content used to create color matched original image, suchas provided for by pyrus color match processing discussed above inreference to FIG. 7. The parametrically linear corrected original imagecontent of FIG. 6 is only globally matched with respect to the to theclassic image content, and therefore is imperfectly color matchedregion-by-region. In FIG. 9, the parametrically linear correctedoriginal image content is regionally color and luminance matched to thecorresponding classic image content using low frequency image contentinformation (e.g., from low pass blur) and high frequency image contentinformation (e.g., from pyrus color match gain computations(s)). Inaddition, a best-fit match can extend into areas in which the classicand original image content does not overlap.

As shown in FIG. 10, in a preferred embodiment, the pyrus color matchgain (block 320 in FIG. 7 includes the low-pass original image content(block 306) and the low-pass classic image content (block 310) havingderivatives thereof (block 402 and 404, respectively) decomposed intorespective horizontal terms and vertical terms. For each pixel, thesquare of the horizontal terms and the square of the vertical terms forthe low-pass original image content derivatives (block 402) are addedtogether (block 406) and the square of the horizontal terms and thesquare of the vertical terms for the low-pass classic image contentderivatives (block 404) are added together (block 408) for each pixel,followed by the resulting images from the classic and derivativesummations each being blurred (block 410, 412 respectively) to produceblurred classic and original derivative image content (block 414, 416,respectively). The square root (block 418) of the ratio between theblurred classic derivative image content and the blurred originalderivative image content is taken to produce the pyrus color match gain(block 322; see also in FIG. 7).

The low pass version of the parametrically match original image (block306) is filtered to produce a bandpass version (block 414) of theparametrically matched original (block 146 in FIG. 7). In the depictedembodiment, derivatives and low pass blurs are used to create themid-frequencies. The low pass version of the classic image (block 310)is filtered to produce a bandpass version (block 416) of the classicimage content (block 148 in FIG. 7). In this embodiment, derivatives andlow pass blurs were used to create the mid-frequencies. The square root(block 418) of the ratio between the classic bandpass and originalbandpass determines the regional gains. Of course standardreasonableness as applied by a person of ordinary skill in the art needsto be taken into account (e.g. no divide by zero).

In view of the disclosures made herein, a skilled person will understandthat the parametric color match provides global adjustment of color ofthe original image content with respect to the overlapping area of theclassic image content and the pyrus color match provides regionaladjustment of color of the original image content. More specifically,the parametric color match is a first color match process that isfollowed by the pyrus color match, which is a second color matchprocess. The parametric color match process adjusts colorcharacteristics of the original image content as a function of a globalcolor match correlation (e.g., global color match regression) derivedfrom the overlapping area of the classic and original image contentsthereby producing a globally color matched version of the original imagecontent. The pyrus color match is a second color match process that isperformed after the first color match process for adjusting colorcharacteristics of each one of a plurality of different regions of theglobally color matched original image content as a function of arespective one of a plurality of regional color match correlations(e.g., regional color match regressions). Each of the regional colormatch correlations is derived from a corresponding region of theoverlapping area of the classic and original image contents.

Advantageously, the pyrus color match allows for color adjustment ofvisual imagery outside of an area of the classic image content in amanner that takes into account a ‘best-approximation’ of the Director'sIntent within the area of the original image content. For example, whena respective position of a particular region of the globally colormatched original image content is located outside of the overlappingarea, the corresponding region to the at least one of the particularregion of the globally color matched version of the original imagecontent is within the overlapping area at a position defined by therespective position of the particular region of the globally colormatched version of the original image content (e.g., positions aresimilar distance from a perimeter boundary of the overlapping area).

Presented now is a discussion of extended range processing (block 154 inFIG. 3). The extended range processing makes use of additionalinformation of the scan of the original image content to create higherdynamic range original image content (extended range image content) thanis available in the classic image content, but without losing theDirector's Intent. More specifically, extended luminance and color isextracted from the original image content for small areas withoutchanging the Director's Intent for large areas. As a result, for largeareas, the color and luminance are the same, but for small areas thatwere saturated or clipped in the classic image content, more informationand brightness is placed into in the pyrus color matched original imagecontent (see FIG. 9) to produce extended range image content. Theextended range image content retains the original director's aestheticas borne in the classic image content, while at the same time has atleast a portion of the resolution, extended color, sparkle, and extendeddynamic range that was captured and coded in the original image content.This extended range image content is a newly created instantiation ofimage content (i.e., UHD-compliant classic image content) characterizedby classic image content that is provided with UHD attributes and thatmaintains the aesthetic characterization defined by the Director'sIntent and, thus, combines desirable imagery features of the classicimage content with desirable imagery features of the correspondingoriginal image content.

Referring now to FIGS. 11A and 11B, in a preferred embodiment of thepresent invention, extended range processing (block 154; see also inFIG. 3) includes pyrus color matched original image content (block 156;see also in FIG. 3) being processed with a surface blur (block 502) thatdoes not exceed the gradient of the input edges, thereby producing asurface blurred version of the pyrus color matched original imagecontent (block 504), which is referred to herein color matched originalimage content DNA (DNA_CM). For example, this surface blur may beimplemented using a “surface blur” tool such as that found in AdobePhotoshop, in which integrated weighting is a function of both distanceand of pixel count level spread. The surface blur contemplated in thisdisclosure is used as a control in many commercial implementations ofshadow fill, such as “Digital SHO” such as sold by Imadio Images, and inthe shadow extension function of Adobe Photoshop.

The pyrus color matched original image content (block 504) exhibits theoverall color for the entire area of the original image content thatmatches the classic image content, as produced from the pyrus colormatched original image content 156. The linear matched original imagecontent (block 150) is used for details and extended range information.The linear matched original image content in the linear region is usedto create the detail for the intensities for that region of intensities.The linear matched original image content outside the linear region isused to create the detail and intensities (i.e., color intensities) forthe saturated intensities regions of the classic image.

Parametrically linear color matched (i.e., linear matched) originalimage content (block 150; see also in FIG. 3) is subjected to abrightness limiting process (block 506), which allows saturatedhighlights to be processed separately. Output of brightness limitingprocess is processed with the aforementioned surface blur (block 508),thereby producing a surface blurred version of the parametrically linearcolor matched (i.e., linear matched) original image content (block 510),which is referred to herein linear color matched original image contentDNA (DNA_LM). The brightness limiting process limits the brightness ofthe parametrically linear color matched original image content to arange equal to or approximately equal to (i.e., derive from) thebrightness of the classic image content.

Output of the brightness limiting process is computationally subtracted(block 514) from the linear color matched original image content DNA(DNA-LM) to produce linear color matched original image contentCytoplasm (Cytoplasm-LM; block 512). Output of the brightness limitingprocess (block 508) is computationally subtracted (block 516) from thelinear color matched original image content, the resulting image contentof which is then subjected to a contrast lowering process (block 518).The contrast lowering process lowers the contrast of very bright regionsof the applied image content. Such applied image content has abrightness above that allowed by the classic image. Ideally, there isalso a roll-off of contrast for very bright areas. Assuming thebrightness allowed by the classic image is 1, the brightness loweringcan be accomplished by the function Ln(X+1), or natural logarithm ofX+1. Output of the contrast lowering process (block 518) is processedwith the aforementioned surface blur (block 520), thereby producing asurface blurred version of the output of the contrast lowering process(block 518), which is referred to herein linear color matched originalimage content DNA highlights (DNA_LM_Highlights). Output of the surfaceblur is computationally subtracted (block 524) from the linear colormatched original image content, thereby producing image content referredto herein as the Highlight Cytoplasm (block 526).

The linear color matched original image content Cytoplasm (block 512)and the Highlight Cytoplasm (block 526) are computationally combined(block 528) to produce Combined Cytoplasm (block 530). The color matchedoriginal image content DNA and the Combined Cytoplasm arecomputationally combined (block 532) to produce recombined DNA+Cytoplasmcolor matched original image content (Recombined DNA+Cytoplasm_CM; block534).

The recombined DNA+Cytoplasm color matched original image content issubjected to a saturated area finding process (block 536) and to aluminance integration process (block 538). The luminance integrationprocess for the recombined DNA+Cytoplasm color matched original imagecontent produces a luminance value (block 540), which is referred toherein as Recombined color matched DNA luminance (luminance_DNA_CM). Theterm luminance as referred to herein is a synonymous with brightness.The parametrically linear color matched original image content (block140) is subject to a luminance integration process (block 542). Theluminance integration process for the parametrically linear colormatched original image content produces a luminance value (block 544),which is referred to herein as linear color matched DNA luminance(luminance_DNA_LM). The Recombined color matched DNA luminance and thelinear color matched DNA luminance are subjected to a luminance ratioprocess (block 546), which produces a luminance ratio (block 548) forthe Recombined color matched DNA luminance with respect to the linearcolor matched DNA luminance. The luminance ratio is applied (block 550),parametric linear matched original image content, which produces themodified linear color match (block 552).

The luminance ratio (block 546) of the recombined color matched image(i.e., color and detail) to the linear matched original image contentdetermines how much of the linear marched original image content to useat each pixel. This modified linear match original image content (block552) is then combined with the recombined color match image content(block 534) to create the extend range image.

The recombined color matched original image content DNA (block 534) andthe modified linear color match (block 552) are subjected to a subjectedto an extended color definition process (block 554) for producing theextended range image content (block 116; see also in FIGS. 1 and 3). Thefunction of the extended color definition process is to combine therecombined DNA+Cytoplasm color matched original image content (block534) and the modified linear color match (block 552) such that the imagecontent of the modified linear match image prevails where saturationoccurs, thereby creating the extended range image content. Therecombined DNA+Cytoplasm color matched original image content is used todetermine where such saturation occurs. FIG. 12 shows a comparison ofthe extended range image content and pyrus color matched original imagecontent.

As discussed above, embodiments of the present invention are directed toan “original” that is matched to a “classic”. However, in certainembodiments of the present invention, image content is limited to an“original”. But such original image content that does not have acorresponding source of corresponding existing classic image content cangreatly benefit from being processed in accordance with all or a portionof the UHD processing disclosed herein. Such processing will retainfeature and accent details (e.g., sparkles) and highlight color whilepreventing large areas (e.g., sky, windows, lights, etc) from blinding aviewer or exceeding the global display power budget of a displayapparatus. To this end, “classic” image content can be derived duringsuch UHD processing from the original image content. For example, a usercould adjust the normal controls of “lift-gamma-gain” of RGB on a scan,as is normally done, to produce a normal adjusted image. That thenbecomes the “classic” for which the UHD processing performs color matchand extended range functionalities to restore highlight colors andfeature and accent details without blinding brightness in large areas.Similarly, the “lift-gamma-gain” could be rendered in automation usingone of many existing autocolor programs. Accordingly, suchimplementation of the present invention expands on what is disclosed asbeing classic image content to include a rendering that is donecontemporaneously with the UHD processing.

The disclosures presented herein have been made in the context of atraditional definition of color (e.g., a full spectrum color such asred, green, blue and combinations thereof). However, embodiments of thepresent invention are not limited to such traditional definitions ofcolor. Embodiments of the present invention are equally applicable toimage content partially of fully comprising ‘grayscale color’. Grayscalecolor preferably refers to different relative proportions of white andblack within a uniform pattern of dots. As such, it is disclosed hereinthat the term ‘color’ can be broadly construed to be traditional coloror grayscale color.

A skilled person will readily understand that the present invention canbe embodied as a system that performs UHD processing in accordance withthe disclosures herein. For example, such a system can be in the form ofa data processing apparatus (computer workstation or the like) coupledto a film scanner and executing instruction (e.g., accessed from anon-transitory computer readable medium) that causes UHD processing tobe formed on classic image content received via a digital networkinterface of the data processing apparatus. An example of the UHDprocessing is discussed herein in reference to FIGS. 1 and 3. Inpreferred embodiments, the UHD processing results in creation of anon-transitory computer readable medium (e.g., storage media, opticaldisc, semiconductor-based memory device, or the like) having tangiblyembodied thereon and accessible therefrom processor-interpretableinformation defining a displayable visual experience (i.e.,UHD-compliant classic image content defining a visual experience such asa feature film). In accordance with UHD processing disclosed herein, theprocessor-interpretable information comprises extended range imagecontent jointly derived from image content of at least one originalimage content source and from image content of a classic image contentsource. The classic image content is a derivative of the original imagecontent. The extend range image content is characterized by extendedrange image content defined by a Director's Intent of the classic imagecontent.

Although the invention has been described with reference to severalexemplary embodiments, it is understood that the words that have beenused are words of description and illustration, rather than words oflimitation. Changes may be made within the purview of the appendedclaims, as presently stated and as amended, without departing from thescope and spirit of the invention in all its aspects. Although theinvention has been described with reference to particular means,materials and embodiments, the invention is not intended to be limitedto the particulars disclosed; rather, the invention extends to allfunctionally equivalent technologies, structures, methods and uses suchas are within the scope of the appended claims.

1. A computer-implemented method, comprising: determining an overlappingarea of image content of at least one original image content source andimage content of a classic image content source in which visual imageryof said classic image content aligns with visual imagery of acorresponding portion of said original image content; performing aparametric color match of said original image content as a function ofsaid classic image content to create linear matched original imagecontent and parametrically color matched original image content, whereinperforming the parametric color match includes matching image contentcolor of said original image content to said classic image content as afunction of a color comparison information derived from said classicimage content and said original image content within the overlappingarea; and performing a pyrus color match using the parametrically colormatched original image content and said classic image content to producepyrus color matched original image content, wherein the pyrus colormatched original image content is characterized by color intensityaltered in accordance with image content information derived from atleast a portion of the image content of the classic image contentsource.
 2. The computer-implemented method of claim 1 wherein: the atleast one original image content source includes a plurality of imagecontent frames each including respective visual imagery; and determiningthe overlapping area includes determining a portion of the classic imagecontent source that has matching visual imagery as a particular one ofthe frames of the at least one original image content source andaligning the visual imagery of the particular one of the frames of theat least one original image content source with the matching visualimagery of the classic image content source.
 3. The computer-implementedmethod of claim 1 wherein the color comparison information includes aregression derived from said classic image content and said originalimage content within the overlapping area.
 4. The computer-implementedmethod of claim 1, further comprising: deriving extended range imagecontent from the pyrus color matched original image content and thelinear matched original image content; wherein deriving extended rangeimage content includes determining a proportionality for unsaturatingcolor in portions of the pyrus color matched original image content thatare within the color saturation range of the classic image content; anddetermining the proportionality for unsaturating color for a particularportion of the pyrus color matched original image content is performedas a function of degree of saturation of the particular portion and arelative size of the particular portion with respect to adjacentportions thereof.
 5. The computer-implemented method of claim 1, furthercomprising: deriving extended range image content from the pyrus colormatched original image content and the linear matched original imagecontent; wherein deriving the extended range image content includesusing the linear matched original image content to determine color inareas of the pyrus color matched original image content that are in thesaturated color condition with respect to a color saturation range ofthe classic image content and to adjust color in at least a portion ofthe areas of the pyrus color matched original image content that are inthe saturated color condition to a color unsaturated condition.
 6. Thecomputer-implemented method of claim 5 wherein the color comparisoninformation includes a regression derived from said classic imagecontent and said original image content within the overlapping area. 7.The computer-implemented method of claim 6 wherein deriving the extendedrange image content includes: using the linear matched original imagecontent in a linear region of the regression for determining colorintensities in the liner region; and using the linear matched originalimage content outside the linear region to determine color intensitiesfor regions of saturated intensity in the classic image content.
 8. Thecomputer-implemented method of claim 1 wherein performing the parametriccolor match to create the parametrically color matched original imagecontent includes determining portions of the linear matched originalimage content having color that is in a color saturation range of saidclassic image content and adjusting a degree of saturation of saidoriginal image content portions as a function of the color saturationrange of said classic image content.
 9. The computer-implemented methodof claim 8, further comprising: deriving extended range image contentfrom the pyrus color matched original image content and the linearmatched original image content; wherein deriving extended range imagecontent includes determining a proportionality for unsaturating color inportions of the pyrus color matched original image content that arewithin the color saturation range of the classic image content; anddetermining the proportionality for unsaturating color for a particularportion of the pyrus color matched original image content is performedas a function of degree of saturation of the particular portion and arelative size of the particular portion with respect to adjacentportions thereof
 10. A method performed for processing image content,comprising: providing first image content and second image content,wherein the second image content is characterized by image contentinformation derived from at least a portion of classic image contentapplied thereto and the second image content is characterized by imagecontent thereof having color attributes altered by said image contentinformation with respect to the first image content and wherein thesecond image content includes at least a portion of visual imagery ofthe first image content; registering the first image content and thesecond image content to defining an overlapping area of the first imagecontent and the second image content in which visual imagery of thesecond image content aligns with visual imagery of a correspondingportion of the first image content; determining aestheticcharacterization defined by said image content information as a functionof the second image content and the portion of the first image contentwithin the overlapping area; and applying the aesthetic characterizationto an entire area of the first image content to create color matchedoriginal image content.
 11. The method of claim 10 wherein: the firstimage content is a selected one of a plurality of frames of imagecontent each including respective visual imagery; and registering thefirst image content and the second image content includes determining aportion of the second image content that has matching visual imagery asthe selected one of the frames of the first image content and aligningthe visual imagery of the selected one of the frames of the first imagecontent with the matching visual imagery of the second image content.12. The method of claim 10 wherein the determining the aestheticcharacterization includes determining a regression derived from thesecond image content and the first image content within the overlappingarea.
 13. The method of claim 10, further comprising: applying extendedrange image content to the color matched original image content; whereinapplying extended range image content includes determining aproportionality for unsaturating color in portions of the color matchedoriginal image content that are within the color saturation range of theclassic image content; and wherein determining the proportionality forunsaturating color for a particular portion of the color matchedoriginal image content is performed as a function of degree ofsaturation of the particular portion and a relative size of theparticular portion with respect to adjacent portions thereof.
 14. Themethod of claim 13, further comprising: applying extended range imagecontent to the color matched original image content; wherein applyingthe extended range image content includes determining color in areas ofthe color matched original image content that are in the saturated colorcondition with respect to a color saturation range of the classic imagecontent and adjusting color in at least a portion of the areas of thecolor matched original image content that are in the saturated colorcondition to a color unsaturated condition.
 15. The method of claim 14wherein applying the aesthetic characterization includes matching imagecontent color of the first image content to image content color of thesecond image content as a function of color comparison informationderived from the second image content and the first image content withinthe overlapping area.
 16. The method of claims 15 wherein the colorcomparison information includes a regression derived from the secondimage content and the first image content within the overlapping area.17. The method of claim 10 wherein applying the aestheticcharacterization includes determining portions of the color matchedoriginal image content having color that is in a color saturation rangeof the second image content and adjusting a degree of saturation of theportions of the color matched original image content as a function ofthe color saturation range of said classic image content.
 18. The methodof claim 17 further comprising: applying extended range image content tothe color matched original image content; wherein applying the extendedrange image content includes determining a proportionality forunsaturating color in portions of the color matched original imagecontent that are within the color saturation range of the classic imagecontent; and wherein determining the proportionality for unsaturatingcolor for a particular portion of the color matched original imagecontent is performed as a function of degree of saturation of theparticular portion and a relative size of the particular portion withrespect to adjacent portions thereof.
 19. The method of claim 10,further comprising: applying extended range image content to the colormatched original image content; wherein applying extended range imagecontent includes using the linear matched original image content in alinear region of the regression for determining color intensities in theliner region and using the linear matched original image content outsidethe linear region to determine color intensities for regions ofsaturated intensity in the classic image content.
 20. (canceled) 21.(canceled)
 22. (canceled)
 23. (canceled)
 24. (canceled)
 25. (canceled)26. (canceled)
 27. (canceled)
 28. (canceled)